Bank Churn Prediction

Problem Statement¶

Context¶

Businesses like banks which provide service have to worry about problem of 'Customer Churn' i.e. customers leaving and joining another service provider. It is important to understand which aspects of the service influence a customer's decision in this regard. Management can concentrate efforts on improvement of service, keeping in mind these priorities.

Objective¶

You as a Data scientist with the bank need to build a neural network based classifier that can determine whether a customer will leave the bank or not in the next 6 months.

Data Dictionary¶

  • CustomerId: Unique ID which is assigned to each customer

  • Surname: Last name of the customer

  • CreditScore: It defines the credit history of the customer.

  • Geography: A customer’s location

  • Gender: It defines the Gender of the customer

  • Age: Age of the customer

  • Tenure: Number of years for which the customer has been with the bank

  • NumOfProducts: refers to the number of products that a customer has purchased through the bank.

  • Balance: Account balance

  • HasCrCard: It is a categorical variable which decides whether the customer has credit card or not.

  • EstimatedSalary: Estimated salary

  • isActiveMember: Is is a categorical variable which decides whether the customer is active member of the bank or not ( Active member in the sense, using bank products regularly, making transactions etc )

  • Exited : whether or not the customer left the bank within six month. It can take two values 0=No ( Customer did not leave the bank ) 1=Yes ( Customer left the bank )

In [ ]:
 

Importing necessary libraries¶

In [ ]:
import pandas as pd  # Library for data manipulation and analysis.
import numpy as np   # Fundamental package for scientific computing.
import matplotlib.pyplot as plt  # Plotting library for creating visualizations.
import seaborn as sns #For advanced visualizations.
import random  # Library for generating random numbers.

from sklearn.model_selection import train_test_split  # Function for splitting datasets for training and testing.
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import confusion_matrix

from imblearn.over_sampling import SMOTE  # Synthetic Minority Over-sampling Technique.

import time  # Module for time-related operations.

import tensorflow as tf #An end-to-end open source machine learning platform
from tensorflow import keras  # High-level neural networks API for deep learning.
from keras import backend   # Abstraction layer for neural network backend engines.
from keras.models import Sequential  # Model for building NN sequentially.
from keras.layers import Dense,Dropout   # for creating fully connected neural network layers.
from tensorflow.keras.optimizers import SGD, Adam

# To suppress warnings
import warnings
warnings.filterwarnings("ignore")
In [ ]:
# Set display options to avoid scientific notation
pd.set_option('display.float_format', '{:.2f}'.format)

# Set the option to display all columns
pd.set_option('display.max_columns', None)

Loading the dataset¶

In [ ]:
# uncomment and run the following line if using Google Colab
from google.colab import drive
drive.mount('/content/drive')
Mounted at /content/drive
In [ ]:
df = pd.read_csv("/content/drive/MyDrive/Personal/UT Austin/Project 4 - NN/Churn.csv")
In [ ]:
data = df.copy()

RowNumber, CustomerID, and Surname won't be used in the analysis, so can remove them.

In [ ]:
#Remove RowNumber, CustomerId and Surname columns from data
data.drop(['RowNumber','CustomerId','Surname'], axis=1, inplace=True)

Data Overview¶

In [ ]:
data.head(5)
Out[ ]:
CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
0 619 France Female 42 2 0.00 1 1 1 101348.88 1
1 608 Spain Female 41 1 83807.86 1 0 1 112542.58 0
2 502 France Female 42 8 159660.80 3 1 0 113931.57 1
3 699 France Female 39 1 0.00 2 0 0 93826.63 0
4 850 Spain Female 43 2 125510.82 1 1 1 79084.10 0
In [ ]:
data.tail(5)
Out[ ]:
CreditScore Geography Gender Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited
9995 771 France Male 39 5 0.00 2 1 0 96270.64 0
9996 516 France Male 35 10 57369.61 1 1 1 101699.77 0
9997 709 France Female 36 7 0.00 1 0 1 42085.58 1
9998 772 Germany Male 42 3 75075.31 2 1 0 92888.52 1
9999 792 France Female 28 4 130142.79 1 1 0 38190.78 0

Checking the shape of the dataset, attribute types, and statistical summary¶

In [ ]:
data.shape
Out[ ]:
(10000, 11)
In [ ]:
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 11 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   CreditScore      10000 non-null  int64  
 1   Geography        10000 non-null  object 
 2   Gender           10000 non-null  object 
 3   Age              10000 non-null  int64  
 4   Tenure           10000 non-null  int64  
 5   Balance          10000 non-null  float64
 6   NumOfProducts    10000 non-null  int64  
 7   HasCrCard        10000 non-null  int64  
 8   IsActiveMember   10000 non-null  int64  
 9   EstimatedSalary  10000 non-null  float64
 10  Exited           10000 non-null  int64  
dtypes: float64(2), int64(7), object(2)
memory usage: 859.5+ KB
In [ ]:
#Get Summary Statistics for the numerical columns
data.describe().T
Out[ ]:
count mean std min 25% 50% 75% max
CreditScore 10000.0 650.528800 96.653299 350.00 584.00 652.000 718.0000 850.00
Age 10000.0 38.921800 10.487806 18.00 32.00 37.000 44.0000 92.00
Tenure 10000.0 5.012800 2.892174 0.00 3.00 5.000 7.0000 10.00
Balance 10000.0 76485.889288 62397.405202 0.00 0.00 97198.540 127644.2400 250898.09
NumOfProducts 10000.0 1.530200 0.581654 1.00 1.00 1.000 2.0000 4.00
HasCrCard 10000.0 0.705500 0.455840 0.00 0.00 1.000 1.0000 1.00
IsActiveMember 10000.0 0.515100 0.499797 0.00 0.00 1.000 1.0000 1.00
EstimatedSalary 10000.0 100090.239881 57510.492818 11.58 51002.11 100193.915 149388.2475 199992.48
Exited 10000.0 0.203700 0.402769 0.00 0.00 0.000 0.0000 1.00

Checking for missing values or duplicate values¶

In [ ]:
# checking for null values
data.isnull().sum()
Out[ ]:
0
CreditScore 0
Geography 0
Gender 0
Age 0
Tenure 0
Balance 0
NumOfProducts 0
HasCrCard 0
IsActiveMember 0
EstimatedSalary 0
Exited 0

No missing values!

In [ ]:
#Checking for duplicate values
data.duplicated().sum()
Out[ ]:
0

No duplicate values!

Checking the mix of customers who left vs stayed with the bank.¶

In [ ]:
# Get counts of each unique value
data['Exited'].value_counts()
Out[ ]:
count
Exited
0 7963
1 2037

In [ ]:
# Get percentages of each unique value
data['Exited'].value_counts(normalize=True) * 100
Out[ ]:
proportion
Exited
0 79.63
1 20.37

Customers who left make up 20% of the total customers, so there is an imbalance the model may need to consider.

Exploratory Data Analysis¶

Some plotting functions we'll use.¶

In [ ]:
# function to plot a boxplot and a histogram along the same scale.


def histogram_boxplot(data, feature, figsize=(12, 7), kde=False, bins=None):
    """
    Boxplot and histogram combined

    data: dataframe
    feature: dataframe column
    figsize: size of figure (default (12,7))
    kde: whether to the show density curve (default False)
    bins: number of bins for histogram (default None)
    """
    f2, (ax_box2, ax_hist2) = plt.subplots(
        nrows=2,  # Number of rows of the subplot grid= 2
        sharex=True,  # x-axis will be shared among all subplots
        gridspec_kw={"height_ratios": (0.25, 0.75)},
        figsize=figsize,
    )  # creating the 2 subplots
    sns.boxplot(
        data=data, x=feature, ax=ax_box2, showmeans=True, color="violet"
    )  # boxplot will be created and a triangle will indicate the mean value of the column
    sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2, bins=bins, palette="winter"
    ) if bins else sns.histplot(
        data=data, x=feature, kde=kde, ax=ax_hist2
    )  # For histogram
    ax_hist2.axvline(
        data[feature].mean(), color="green", linestyle="--"
    )  # Add mean to the histogram
    ax_hist2.axvline(
        data[feature].median(), color="black", linestyle="-"
    )  # Add median to the histogram
In [ ]:
# function to create labeled barplots


def labeled_barplot(data, feature, perc=False, n=None):
    """
    Barplot with percentage at the top

    data: dataframe
    feature: dataframe column
    perc: whether to display percentages instead of count (default is False)
    n: displays the top n category levels (default is None, i.e., display all levels)
    """

    total = len(data[feature])  # length of the column
    count = data[feature].nunique()
    if n is None:
        plt.figure(figsize=(count + 1, 5))
    else:
        plt.figure(figsize=(n + 1, 5))

    plt.xticks(rotation=90, fontsize=15)
    ax = sns.countplot(
        data=data,
        x=feature,
        palette="Paired",
        order=data[feature].value_counts().index[:n].sort_values(),
    )

    for p in ax.patches:
        if perc == True:
            label = "{:.1f}%".format(
                100 * p.get_height() / total
            )  # percentage of each class of the category
        else:
            label = p.get_height()  # count of each level of the category

        x = p.get_x() + p.get_width() / 2  # width of the plot
        y = p.get_height()  # height of the plot

        ax.annotate(
            label,
            (x, y),
            ha="center",
            va="center",
            size=12,
            xytext=(0, 5),
            textcoords="offset points",
        )  # annotate the percentage

    plt.show()  # show the plot
In [ ]:
# function to plot stacked bar chart

def stacked_barplot(data, predictor, target):
    """
    Print the category counts and plot a stacked bar chart

    data: dataframe
    predictor: independent variable
    target: target variable
    """
    count = data[predictor].nunique()
    sorter = data[target].value_counts().index[-1]
    tab1 = pd.crosstab(data[predictor], data[target], margins=True).sort_values(
        by=sorter, ascending=False
    )
    print(tab1)
    print("-" * 120)
    tab = pd.crosstab(data[predictor], data[target], normalize="index").sort_values(
        by=sorter, ascending=False
    )
    tab.plot(kind="bar", stacked=True, figsize=(count + 1, 5))
    plt.legend(
        loc="lower left", frameon=False,
    )
    plt.legend(loc="upper left", bbox_to_anchor=(1, 1))
    plt.show()
In [ ]:
### Function to plot distributions

def distribution_plot_wrt_target(data, predictor, target):

    fig, axs = plt.subplots(2, 2, figsize=(12, 10))

    target_uniq = data[target].unique()

    axs[0, 0].set_title("Distribution of target for target=" + str(target_uniq[0]))
    sns.histplot(
        data=data[data[target] == target_uniq[0]],
        x=predictor,
        kde=True,
        ax=axs[0, 0],
        color="teal",
    )

    axs[0, 1].set_title("Distribution of target for target=" + str(target_uniq[1]))
    sns.histplot(
        data=data[data[target] == target_uniq[1]],
        x=predictor,
        kde=True,
        ax=axs[0, 1],
        color="orange",
    )

    axs[1, 0].set_title("Boxplot w.r.t target")
    sns.boxplot(data=data, x=target, y=predictor, ax=axs[1, 0], palette="gist_rainbow")

    axs[1, 1].set_title("Boxplot (without outliers) w.r.t target")
    sns.boxplot(
        data=data,
        x=target,
        y=predictor,
        ax=axs[1, 1],
        showfliers=False,
        palette="gist_rainbow",
    )

    plt.tight_layout()
    plt.show()

Univariate Analysis¶

Numerical Variables¶

In [ ]:
histogram_boxplot(data, 'CreditScore')
In [ ]:
histogram_boxplot(data, 'Age')
In [ ]:
histogram_boxplot(data, 'Tenure')
In [ ]:
histogram_boxplot(data, 'Balance')
In [ ]:
histogram_boxplot(data, 'NumOfProducts')
In [ ]:
histogram_boxplot(data, 'EstimatedSalary')

Categorical Variables¶

In [ ]:
labeled_barplot(data, 'Gender')
In [ ]:
labeled_barplot(data, 'Geography')
In [ ]:
labeled_barplot(data, 'HasCrCard')
In [ ]:
labeled_barplot(data, 'IsActiveMember')

Bivariate Analysis¶

Let's look at the correlation matrix for all the numeric variables.

In [ ]:
# Exclude specific columns from the correlation matrix
excluded_columns = ['Exited', 'IsActiveMember', 'HasCrCard']
filtered_data = data.drop(columns=excluded_columns)

# plotting the correlation heatmap
plt.figure(figsize=(12, 7))
sns.heatmap(filtered_data.corr(numeric_only=True), annot=True, fmt='0.2f', cmap='Spectral')
plt.show()

From the above we can see that there is almost no correlation between the numeric variables.

In [ ]:
sns.pairplot(data,hue='Exited')
plt.show()

Numerical Variables¶

In [ ]:
distribution_plot_wrt_target(data, 'CreditScore', 'Exited')
In [ ]:
distribution_plot_wrt_target(data, 'Age', 'Exited')
In [ ]:
distribution_plot_wrt_target(data, 'Tenure', 'Exited')
In [ ]:
distribution_plot_wrt_target(data, 'Balance', 'Exited')
In [ ]:
distribution_plot_wrt_target(data, 'NumOfProducts', 'Exited')
In [ ]:
distribution_plot_wrt_target(data, 'EstimatedSalary', 'Exited')

Categorical Variables¶

In [ ]:
stacked_barplot(data, 'Gender', 'Exited')
Exited     0     1    All
Gender                   
All     7963  2037  10000
Female  3404  1139   4543
Male    4559   898   5457
------------------------------------------------------------------------------------------------------------------------
In [ ]:
stacked_barplot(data, 'Geography', 'Exited')
Exited        0     1    All
Geography                   
All        7963  2037  10000
Germany    1695   814   2509
France     4204   810   5014
Spain      2064   413   2477
------------------------------------------------------------------------------------------------------------------------
In [ ]:
stacked_barplot(data, 'HasCrCard', 'Exited')
Exited        0     1    All
HasCrCard                   
All        7963  2037  10000
1          5631  1424   7055
0          2332   613   2945
------------------------------------------------------------------------------------------------------------------------
In [ ]:
stacked_barplot(data, 'IsActiveMember', 'Exited')
Exited             0     1    All
IsActiveMember                   
All             7963  2037  10000
0               3547  1302   4849
1               4416   735   5151
------------------------------------------------------------------------------------------------------------------------

Data Preprocessing¶

Dummy Variable Creation¶

Lets convert the columns with an 'object' datatype into categorical variables, to reduce the space required to store the dataframe and improve processing efficieny.

In [ ]:
for feature in data.columns: # Loop through all columns in the dataframe
    if data[feature].dtype == 'object': # Only apply for columns with categorical strings
        data[feature] = pd.Categorical(data[feature]) # Convert the column to a categorical type
In [ ]:
# Confirm that column of type object have been converted to category
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 11 columns):
 #   Column           Non-Null Count  Dtype   
---  ------           --------------  -----   
 0   CreditScore      10000 non-null  int64   
 1   Geography        10000 non-null  category
 2   Gender           10000 non-null  category
 3   Age              10000 non-null  int64   
 4   Tenure           10000 non-null  int64   
 5   Balance          10000 non-null  float64 
 6   NumOfProducts    10000 non-null  int64   
 7   HasCrCard        10000 non-null  int64   
 8   IsActiveMember   10000 non-null  int64   
 9   EstimatedSalary  10000 non-null  float64 
 10  Exited           10000 non-null  int64   
dtypes: category(2), float64(2), int64(7)
memory usage: 723.0 KB
In [ ]:
replaceStruct = {
                "Gender": {"M": 0, "F":1},
                 "Geography": {"Germany": 0, "Spain":1 , "France": 2},
                }
oneHotCols=["Gender","Geography"]
In [ ]:
data=data.replace(replaceStruct)
data=pd.get_dummies(data, columns=oneHotCols, drop_first=True)
data.head(10)
Out[ ]:
CreditScore Age Tenure Balance NumOfProducts HasCrCard IsActiveMember EstimatedSalary Exited Gender_Male Geography_0 Geography_1
0 619 42 2 0.00 1 1 1 101348.88 1 False False False
1 608 41 1 83807.86 1 0 1 112542.58 0 False False True
2 502 42 8 159660.80 3 1 0 113931.57 1 False False False
3 699 39 1 0.00 2 0 0 93826.63 0 False False False
4 850 43 2 125510.82 1 1 1 79084.10 0 False False True
5 645 44 8 113755.78 2 1 0 149756.71 1 True False True
6 822 50 7 0.00 2 1 1 10062.80 0 True False False
7 376 29 4 115046.74 4 1 0 119346.88 1 False True False
8 501 44 4 142051.07 2 0 1 74940.50 0 True False False
9 684 27 2 134603.88 1 1 1 71725.73 0 True False False
In [ ]:
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10000 entries, 0 to 9999
Data columns (total 12 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   CreditScore      10000 non-null  int64  
 1   Age              10000 non-null  int64  
 2   Tenure           10000 non-null  int64  
 3   Balance          10000 non-null  float64
 4   NumOfProducts    10000 non-null  int64  
 5   HasCrCard        10000 non-null  int64  
 6   IsActiveMember   10000 non-null  int64  
 7   EstimatedSalary  10000 non-null  float64
 8   Exited           10000 non-null  int64  
 9   Gender_Male      10000 non-null  bool   
 10  Geography_0      10000 non-null  bool   
 11  Geography_1      10000 non-null  bool   
dtypes: bool(3), float64(2), int64(7)
memory usage: 732.5 KB

Train-validation-test Split¶

In [ ]:
X = data.drop(columns=['Exited'])  # Features (all except the target 'Exited')
y = data['Exited']                 # Target

# Step 1: Split into 80% train+validation and 20% test
X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# Step 2: Split the 80% train+validation into 75% train and 25% validation
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=0.25, random_state=42, stratify=y_train_val)

# Resulting splits:
# X_train, y_train -> 60% of the data (training set)
# X_val, y_val -> 20% of the data (validation set)
# X_test, y_test -> 20% of the data (test set)

# Print the shapes of the resulting datasets
print("Training set shape:", X_train.shape, y_train.shape)
print("Validation set shape:", X_val.shape, y_val.shape)
print("Test set shape:", X_test.shape, y_test.shape)
Training set shape: (6000, 11) (6000,)
Validation set shape: (2000, 11) (2000,)
Test set shape: (2000, 11) (2000,)

Data Normalization¶

Now, we will perform scaling on the numerical variables separately for train, validation, and test sets.

In [ ]:
scaler = MinMaxScaler()

# Here, we are passing all the features (numerical and categorical), that's okay as min-max scaler will not change values of categorical variables
X_train = scaler.fit_transform(X_train)
In [ ]:
X_val = scaler.transform(X_val)
X_test = scaler.transform(X_test)

Utility functions¶

In [ ]:
def plot(history, name, y_min=None, y_max=None):
    """
    Function to plot loss/accuracy

    history: an object which stores the metrics and losses.
    name: can be one of Loss or Accuracy
    y_min: Minimum value for y-axis (optional)
    y_max: Maximum value for y-axis (optional)
    """
    fig, ax = plt.subplots()  # Creating a subplot with figure and axes.
    plt.plot(history.history[name])  # Plotting the train accuracy or train loss
    plt.plot(history.history['val_' + name])  # Plotting the validation accuracy or validation loss

    plt.title('Model ' + name.capitalize())  # Defining the title of the plot.
    plt.ylabel(name.capitalize())  # Capitalizing the first letter.
    plt.xlabel('Epoch')  # Defining the label for the x-axis.
    fig.legend(['Train', 'Validation'], loc="upper right")  # Defining the legend, loc controls the position of the legend.

    # Setting y-axis limits if provided
    if y_min is not None and y_max is not None:
        plt.ylim(y_min, y_max)

    plt.show()

We'll create a dataframe to store the results from all the models we build

In [ ]:
#Defining the columns of the dataframe which are nothing but the hyper parameters and the metrics.
columns = ["model label","# hidden layers","# neurons - hidden layer","activation function - hidden layer","# epochs","batch size","train loss","validation loss","train accuracy","validation accuracy","train recall","validation recall","time (secs)"]

#Creating a pandas dataframe.
results = pd.DataFrame(columns=columns)
In [ ]:
# Function to plot confusion matrix with percentages and metrics
def plot_confusion_matrix(model, predictors, target):
    """
    To plot the confusion_matrix with percentages and additional metrics
    model: classifier
    predictors: independent variables
    target: dependent variable
    """
    from sklearn.metrics import confusion_matrix
    import seaborn as sns

    # Predict the target values using the provided model and predictors
    y_pred = (model.predict(predictors, verbose=0) > 0.5).astype(int)
    # Compute the confusion matrix comparing the true target values with the predicted values
    cm = confusion_matrix(target, y_pred)
    # Create labels for each cell in the confusion matrix with both count and percentage
    labels = np.asarray(
        [
            ["{0:0.0f}".format(item) + "\n{0:.2%}".format(item / cm.sum())]
            for item in cm.flatten()
        ]
    ).reshape(2, 2)  # reshaping to a matrix
    # Set the figure size for the plot
    plt.figure(figsize=(8, 6))
    # Plot the confusion matrix as a heatmap with the labels
    sns.heatmap(cm, annot=labels, fmt="", cmap='Blues')
    # Add a label to the y-axis
    plt.ylabel("True label")
    # Add a label to the x-axis
    plt.xlabel("Predicted label")

    # Compute and display additional metrics
    accuracy = np.trace(cm) / float(np.sum(cm))
    if len(cm) == 2:
        precision = cm[1, 1] / sum(cm[:, 1])
        recall = cm[1, 1] / sum(cm[1, :])
        f1 = 2 * precision * recall / (precision + recall)
        stats_text = "\n\nAccuracy={:0.2f}\nPrecision={:0.2f}\nRecall={:0.2f}\nF1 Score={:0.2f}".format(
            accuracy, precision, recall, f1
        )
    else:
        stats_text = "\n\nAccuracy={:0.2f}".format(accuracy)

    # Set the title and additional metrics
    plt.xlabel("Predicted label" + stats_text)
    plt.title("Confusion Matrix")
    plt.show()

Model Building¶

Model Evaluation Criterion¶

Since we are trying to identify those customer who are at risk of churning and we want to not miss any of the potential ones, therefore, Recall is good metric to use. However, since Exited is imbalanced (80/20) Recall can pose problems as it doesn't account for false-positives. This is where use of SMOTE will come in handy to deal with the class imbalance.

Set the seed for random number generators in NumPy, Python, and TensorFlow to be able to reproduce the same results everytime we run the code.

In [ ]:
# Fixing the seed for random number generators
np.random.seed(42)

random.seed(42)

tf.random.set_seed(42)

smote = SMOTE(random_state=42)

Neural Network with SGD Optimizer¶

We'll start out with a baseline model having the following configuration:

  • 1 input, 2 hidden, 1 output layers
  • relu activations for the hidden layers
  • Stochastic Gradient Descent (SGD)
In [ ]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [ ]:
#Initializing the neural network
model_0 = Sequential()
model_0.add(Dense(64, activation="relu",input_dim = X_train.shape[1])) #Hidden & Input Layer
model_0.add(Dense(32,activation="relu")) #Hidden Layer
model_0.add(Dense(1,activation = "sigmoid")) #Output Layer
In [ ]:
#Printing the summary.
model_0.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             768 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
 Total params: 2,881 (11.25 KB)
 Trainable params: 2,881 (11.25 KB)
 Non-trainable params: 0 (0.00 B)
In [ ]:
optimizer = keras.optimizers.SGD(learning_rate=0.005, momentum=0.9)    # defining SGD as the optimizer to be used
model_0.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy', tf.keras.metrics.Recall(name='recall')])
In [ ]:
#batch_size = X_train.shape[0]
batch_size = 64
epochs = 100

start = time.time()
history = model_0.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.7285 - loss: 0.5957 - recall: 0.1896 - val_accuracy: 0.7965 - val_loss: 0.4898 - val_recall: 0.0000e+00
Epoch 2/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7979 - loss: 0.4839 - recall: 0.0000e+00 - val_accuracy: 0.7965 - val_loss: 0.4825 - val_recall: 0.0000e+00
Epoch 3/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7979 - loss: 0.4763 - recall: 0.0000e+00 - val_accuracy: 0.7965 - val_loss: 0.4770 - val_recall: 0.0000e+00
Epoch 4/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7979 - loss: 0.4704 - recall: 0.0000e+00 - val_accuracy: 0.7965 - val_loss: 0.4732 - val_recall: 0.0000e+00
Epoch 5/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7979 - loss: 0.4659 - recall: 0.0000e+00 - val_accuracy: 0.7965 - val_loss: 0.4699 - val_recall: 0.0000e+00
Epoch 6/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7979 - loss: 0.4620 - recall: 0.0000e+00 - val_accuracy: 0.7965 - val_loss: 0.4669 - val_recall: 0.0000e+00
Epoch 7/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.7979 - loss: 0.4585 - recall: 0.0000e+00 - val_accuracy: 0.7965 - val_loss: 0.4638 - val_recall: 0.0000e+00
Epoch 8/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7981 - loss: 0.4553 - recall: 0.0016 - val_accuracy: 0.7960 - val_loss: 0.4607 - val_recall: 0.0000e+00
Epoch 9/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7998 - loss: 0.4520 - recall: 0.0124 - val_accuracy: 0.7995 - val_loss: 0.4576 - val_recall: 0.0197
Epoch 10/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8042 - loss: 0.4489 - recall: 0.0377 - val_accuracy: 0.8005 - val_loss: 0.4542 - val_recall: 0.0319
Epoch 11/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8039 - loss: 0.4457 - recall: 0.0488 - val_accuracy: 0.8010 - val_loss: 0.4507 - val_recall: 0.0467
Epoch 12/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8056 - loss: 0.4426 - recall: 0.0663 - val_accuracy: 0.8020 - val_loss: 0.4474 - val_recall: 0.0639
Epoch 13/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8083 - loss: 0.4395 - recall: 0.0858 - val_accuracy: 0.8030 - val_loss: 0.4441 - val_recall: 0.0811
Epoch 14/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8119 - loss: 0.4364 - recall: 0.1144 - val_accuracy: 0.8065 - val_loss: 0.4411 - val_recall: 0.0983
Epoch 15/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8158 - loss: 0.4337 - recall: 0.1375 - val_accuracy: 0.8075 - val_loss: 0.4383 - val_recall: 0.1106
Epoch 16/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8191 - loss: 0.4312 - recall: 0.1618 - val_accuracy: 0.8110 - val_loss: 0.4359 - val_recall: 0.1278
Epoch 17/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8176 - loss: 0.4290 - recall: 0.1696 - val_accuracy: 0.8115 - val_loss: 0.4336 - val_recall: 0.1499
Epoch 18/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8169 - loss: 0.4271 - recall: 0.1792 - val_accuracy: 0.8120 - val_loss: 0.4317 - val_recall: 0.1622
Epoch 19/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8169 - loss: 0.4254 - recall: 0.1950 - val_accuracy: 0.8140 - val_loss: 0.4302 - val_recall: 0.1744
Epoch 20/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8181 - loss: 0.4239 - recall: 0.2063 - val_accuracy: 0.8140 - val_loss: 0.4288 - val_recall: 0.1818
Epoch 21/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8185 - loss: 0.4225 - recall: 0.2134 - val_accuracy: 0.8140 - val_loss: 0.4277 - val_recall: 0.1867
Epoch 22/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8204 - loss: 0.4212 - recall: 0.2242 - val_accuracy: 0.8165 - val_loss: 0.4265 - val_recall: 0.2015
Epoch 23/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8197 - loss: 0.4201 - recall: 0.2304 - val_accuracy: 0.8180 - val_loss: 0.4254 - val_recall: 0.2088
Epoch 24/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8191 - loss: 0.4188 - recall: 0.2326 - val_accuracy: 0.8195 - val_loss: 0.4243 - val_recall: 0.2211
Epoch 25/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8187 - loss: 0.4175 - recall: 0.2343 - val_accuracy: 0.8195 - val_loss: 0.4232 - val_recall: 0.2236
Epoch 26/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8191 - loss: 0.4160 - recall: 0.2396 - val_accuracy: 0.8205 - val_loss: 0.4220 - val_recall: 0.2285
Epoch 27/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8192 - loss: 0.4145 - recall: 0.2419 - val_accuracy: 0.8210 - val_loss: 0.4208 - val_recall: 0.2334
Epoch 28/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8207 - loss: 0.4130 - recall: 0.2485 - val_accuracy: 0.8215 - val_loss: 0.4196 - val_recall: 0.2359
Epoch 29/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8217 - loss: 0.4114 - recall: 0.2551 - val_accuracy: 0.8215 - val_loss: 0.4182 - val_recall: 0.2359
Epoch 30/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8240 - loss: 0.4096 - recall: 0.2646 - val_accuracy: 0.8225 - val_loss: 0.4164 - val_recall: 0.2432
Epoch 31/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8242 - loss: 0.4075 - recall: 0.2682 - val_accuracy: 0.8230 - val_loss: 0.4146 - val_recall: 0.2482
Epoch 32/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8259 - loss: 0.4053 - recall: 0.2740 - val_accuracy: 0.8250 - val_loss: 0.4128 - val_recall: 0.2629
Epoch 33/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8264 - loss: 0.4030 - recall: 0.2785 - val_accuracy: 0.8255 - val_loss: 0.4109 - val_recall: 0.2727
Epoch 34/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8285 - loss: 0.4007 - recall: 0.2854 - val_accuracy: 0.8265 - val_loss: 0.4086 - val_recall: 0.2826
Epoch 35/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8292 - loss: 0.3980 - recall: 0.2869 - val_accuracy: 0.8285 - val_loss: 0.4061 - val_recall: 0.2924
Epoch 36/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8318 - loss: 0.3952 - recall: 0.2991 - val_accuracy: 0.8295 - val_loss: 0.4034 - val_recall: 0.2948
Epoch 37/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8346 - loss: 0.3924 - recall: 0.3037 - val_accuracy: 0.8310 - val_loss: 0.4005 - val_recall: 0.3022
Epoch 38/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8354 - loss: 0.3895 - recall: 0.3087 - val_accuracy: 0.8305 - val_loss: 0.3975 - val_recall: 0.3047
Epoch 39/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8379 - loss: 0.3863 - recall: 0.3192 - val_accuracy: 0.8320 - val_loss: 0.3944 - val_recall: 0.3145
Epoch 40/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8396 - loss: 0.3830 - recall: 0.3310 - val_accuracy: 0.8375 - val_loss: 0.3910 - val_recall: 0.3415
Epoch 41/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8434 - loss: 0.3797 - recall: 0.3433 - val_accuracy: 0.8385 - val_loss: 0.3876 - val_recall: 0.3464
Epoch 42/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8444 - loss: 0.3765 - recall: 0.3473 - val_accuracy: 0.8405 - val_loss: 0.3844 - val_recall: 0.3612
Epoch 43/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8457 - loss: 0.3733 - recall: 0.3592 - val_accuracy: 0.8415 - val_loss: 0.3811 - val_recall: 0.3710
Epoch 44/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8460 - loss: 0.3702 - recall: 0.3647 - val_accuracy: 0.8450 - val_loss: 0.3780 - val_recall: 0.3907
Epoch 45/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8472 - loss: 0.3672 - recall: 0.3693 - val_accuracy: 0.8460 - val_loss: 0.3749 - val_recall: 0.3931
Epoch 46/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8503 - loss: 0.3643 - recall: 0.3736 - val_accuracy: 0.8475 - val_loss: 0.3723 - val_recall: 0.3931
Epoch 47/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8516 - loss: 0.3616 - recall: 0.3797 - val_accuracy: 0.8500 - val_loss: 0.3696 - val_recall: 0.4005
Epoch 48/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8524 - loss: 0.3592 - recall: 0.3865 - val_accuracy: 0.8510 - val_loss: 0.3674 - val_recall: 0.4029
Epoch 49/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8525 - loss: 0.3570 - recall: 0.3927 - val_accuracy: 0.8510 - val_loss: 0.3654 - val_recall: 0.4029
Epoch 50/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8534 - loss: 0.3550 - recall: 0.3982 - val_accuracy: 0.8510 - val_loss: 0.3636 - val_recall: 0.4054
Epoch 51/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8541 - loss: 0.3532 - recall: 0.4027 - val_accuracy: 0.8530 - val_loss: 0.3620 - val_recall: 0.4152
Epoch 52/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8538 - loss: 0.3516 - recall: 0.4034 - val_accuracy: 0.8520 - val_loss: 0.3607 - val_recall: 0.4103
Epoch 53/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8549 - loss: 0.3502 - recall: 0.4108 - val_accuracy: 0.8520 - val_loss: 0.3596 - val_recall: 0.4103
Epoch 54/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8559 - loss: 0.3490 - recall: 0.4198 - val_accuracy: 0.8540 - val_loss: 0.3586 - val_recall: 0.4226
Epoch 55/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8562 - loss: 0.3478 - recall: 0.4229 - val_accuracy: 0.8545 - val_loss: 0.3577 - val_recall: 0.4251
Epoch 56/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8563 - loss: 0.3468 - recall: 0.4272 - val_accuracy: 0.8560 - val_loss: 0.3569 - val_recall: 0.4300
Epoch 57/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8557 - loss: 0.3459 - recall: 0.4267 - val_accuracy: 0.8555 - val_loss: 0.3563 - val_recall: 0.4300
Epoch 58/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8563 - loss: 0.3451 - recall: 0.4309 - val_accuracy: 0.8550 - val_loss: 0.3557 - val_recall: 0.4275
Epoch 59/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8555 - loss: 0.3444 - recall: 0.4314 - val_accuracy: 0.8550 - val_loss: 0.3553 - val_recall: 0.4349
Epoch 60/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8564 - loss: 0.3436 - recall: 0.4319 - val_accuracy: 0.8540 - val_loss: 0.3549 - val_recall: 0.4300
Epoch 61/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8563 - loss: 0.3430 - recall: 0.4299 - val_accuracy: 0.8525 - val_loss: 0.3545 - val_recall: 0.4251
Epoch 62/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8568 - loss: 0.3425 - recall: 0.4322 - val_accuracy: 0.8535 - val_loss: 0.3542 - val_recall: 0.4324
Epoch 63/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8570 - loss: 0.3421 - recall: 0.4320 - val_accuracy: 0.8535 - val_loss: 0.3537 - val_recall: 0.4324
Epoch 64/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8576 - loss: 0.3416 - recall: 0.4325 - val_accuracy: 0.8540 - val_loss: 0.3535 - val_recall: 0.4349
Epoch 65/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8571 - loss: 0.3410 - recall: 0.4327 - val_accuracy: 0.8550 - val_loss: 0.3532 - val_recall: 0.4373
Epoch 66/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8570 - loss: 0.3405 - recall: 0.4331 - val_accuracy: 0.8555 - val_loss: 0.3529 - val_recall: 0.4373
Epoch 67/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8583 - loss: 0.3401 - recall: 0.4332 - val_accuracy: 0.8550 - val_loss: 0.3527 - val_recall: 0.4349
Epoch 68/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8577 - loss: 0.3398 - recall: 0.4381 - val_accuracy: 0.8555 - val_loss: 0.3527 - val_recall: 0.4324
Epoch 69/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8586 - loss: 0.3394 - recall: 0.4417 - val_accuracy: 0.8550 - val_loss: 0.3525 - val_recall: 0.4324
Epoch 70/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8595 - loss: 0.3390 - recall: 0.4465 - val_accuracy: 0.8550 - val_loss: 0.3523 - val_recall: 0.4324
Epoch 71/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8592 - loss: 0.3388 - recall: 0.4459 - val_accuracy: 0.8540 - val_loss: 0.3522 - val_recall: 0.4251
Epoch 72/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8595 - loss: 0.3384 - recall: 0.4472 - val_accuracy: 0.8545 - val_loss: 0.3521 - val_recall: 0.4275
Epoch 73/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8594 - loss: 0.3380 - recall: 0.4482 - val_accuracy: 0.8545 - val_loss: 0.3519 - val_recall: 0.4300
Epoch 74/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8591 - loss: 0.3379 - recall: 0.4486 - val_accuracy: 0.8540 - val_loss: 0.3518 - val_recall: 0.4300
Epoch 75/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8588 - loss: 0.3377 - recall: 0.4473 - val_accuracy: 0.8555 - val_loss: 0.3515 - val_recall: 0.4349
Epoch 76/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8599 - loss: 0.3374 - recall: 0.4516 - val_accuracy: 0.8560 - val_loss: 0.3513 - val_recall: 0.4373
Epoch 77/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.8603 - loss: 0.3370 - recall: 0.4541 - val_accuracy: 0.8565 - val_loss: 0.3510 - val_recall: 0.4373
Epoch 78/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8605 - loss: 0.3370 - recall: 0.4564 - val_accuracy: 0.8555 - val_loss: 0.3510 - val_recall: 0.4373
Epoch 79/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8602 - loss: 0.3368 - recall: 0.4545 - val_accuracy: 0.8550 - val_loss: 0.3509 - val_recall: 0.4324
Epoch 80/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8613 - loss: 0.3364 - recall: 0.4580 - val_accuracy: 0.8550 - val_loss: 0.3510 - val_recall: 0.4324
Epoch 81/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.8620 - loss: 0.3361 - recall: 0.4593 - val_accuracy: 0.8540 - val_loss: 0.3509 - val_recall: 0.4324
Epoch 82/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8612 - loss: 0.3360 - recall: 0.4593 - val_accuracy: 0.8540 - val_loss: 0.3509 - val_recall: 0.4275
Epoch 83/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8618 - loss: 0.3357 - recall: 0.4606 - val_accuracy: 0.8555 - val_loss: 0.3507 - val_recall: 0.4324
Epoch 84/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8619 - loss: 0.3355 - recall: 0.4610 - val_accuracy: 0.8550 - val_loss: 0.3507 - val_recall: 0.4324
Epoch 85/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8624 - loss: 0.3352 - recall: 0.4623 - val_accuracy: 0.8545 - val_loss: 0.3506 - val_recall: 0.4324
Epoch 86/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8630 - loss: 0.3349 - recall: 0.4633 - val_accuracy: 0.8560 - val_loss: 0.3507 - val_recall: 0.4324
Epoch 87/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8628 - loss: 0.3348 - recall: 0.4625 - val_accuracy: 0.8545 - val_loss: 0.3504 - val_recall: 0.4324
Epoch 88/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8629 - loss: 0.3346 - recall: 0.4610 - val_accuracy: 0.8550 - val_loss: 0.3506 - val_recall: 0.4324
Epoch 89/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8635 - loss: 0.3344 - recall: 0.4654 - val_accuracy: 0.8545 - val_loss: 0.3505 - val_recall: 0.4324
Epoch 90/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8630 - loss: 0.3341 - recall: 0.4628 - val_accuracy: 0.8560 - val_loss: 0.3505 - val_recall: 0.4300
Epoch 91/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8628 - loss: 0.3338 - recall: 0.4642 - val_accuracy: 0.8560 - val_loss: 0.3506 - val_recall: 0.4300
Epoch 92/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8630 - loss: 0.3337 - recall: 0.4651 - val_accuracy: 0.8560 - val_loss: 0.3504 - val_recall: 0.4300
Epoch 93/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8628 - loss: 0.3335 - recall: 0.4645 - val_accuracy: 0.8565 - val_loss: 0.3505 - val_recall: 0.4300
Epoch 94/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8628 - loss: 0.3334 - recall: 0.4638 - val_accuracy: 0.8570 - val_loss: 0.3504 - val_recall: 0.4324
Epoch 95/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8627 - loss: 0.3331 - recall: 0.4640 - val_accuracy: 0.8580 - val_loss: 0.3503 - val_recall: 0.4349
Epoch 96/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8622 - loss: 0.3329 - recall: 0.4652 - val_accuracy: 0.8575 - val_loss: 0.3503 - val_recall: 0.4324
Epoch 97/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8628 - loss: 0.3327 - recall: 0.4655 - val_accuracy: 0.8575 - val_loss: 0.3503 - val_recall: 0.4300
Epoch 98/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8632 - loss: 0.3324 - recall: 0.4674 - val_accuracy: 0.8565 - val_loss: 0.3503 - val_recall: 0.4324
Epoch 99/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8632 - loss: 0.3323 - recall: 0.4702 - val_accuracy: 0.8570 - val_loss: 0.3503 - val_recall: 0.4349
Epoch 100/100
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8628 - loss: 0.3321 - recall: 0.4686 - val_accuracy: 0.8570 - val_loss: 0.3504 - val_recall: 0.4349
In [ ]:
print("Time taken in seconds ",end-start)
Time taken in seconds  42.23354721069336
In [ ]:
plot(history,'loss')
In [ ]:
plot(history,'accuracy')
In [ ]:
plot(history,'recall')
#print(history.history.keys())
In [ ]:
results.loc[0] = ['NN with SGD',2,'[64,32]','relu',epochs,batch_size,history.history["loss"][-1],history.history["val_loss"][-1],history.history["accuracy"][-1],history.history["val_accuracy"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]

results
Out[ ]:
model label # hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size train loss validation loss train accuracy validation accuracy train recall validation recall time (secs)
0 NN with SGD 2 [64,32] relu 100 64 0.33 0.35 0.86 0.86 0.47 0.43 42.23

Commentary: This model has a good balance between training and validation, suggesting limited overfitting, however, the recall score is not very good. I also tried using tanh in the second hidden layer, however, the peformance was not significantly different, so stayed with using relu in both hidden layers.

Model Performance Improvement¶

Neural Network with Adam Optimizer¶

In [ ]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [ ]:
#Initializing the neural network
model_1 = Sequential()
model_1.add(Dense(64, activation="relu",input_dim = X_train.shape[1])) #Hidden & Input Layer
model_1.add(Dense(32,activation="relu")) #Hidden Layer
model_1.add(Dense(1,activation = "sigmoid")) #Output Layer
In [ ]:
#Printing the summary.
model_1.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             768 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
 Total params: 2,881 (11.25 KB)
 Trainable params: 2,881 (11.25 KB)
 Non-trainable params: 0 (0.00 B)
In [ ]:
optimizer = tf.keras.optimizers.Adam(learning_rate=0.005)    # defining Adam as the optimizer to be used
model_1.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy', tf.keras.metrics.Recall(name='recall')])
In [ ]:
batch_size = 64
epochs = 40

start = time.time()
history = model_1.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 2s 9ms/step - accuracy: 0.7160 - loss: 0.5444 - recall: 0.1123 - val_accuracy: 0.8015 - val_loss: 0.4504 - val_recall: 0.0442
Epoch 2/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8127 - loss: 0.4348 - recall: 0.1266 - val_accuracy: 0.8230 - val_loss: 0.4228 - val_recall: 0.3464
Epoch 3/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8228 - loss: 0.4121 - recall: 0.2707 - val_accuracy: 0.8305 - val_loss: 0.3986 - val_recall: 0.4128
Epoch 4/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8416 - loss: 0.3870 - recall: 0.3750 - val_accuracy: 0.8375 - val_loss: 0.3766 - val_recall: 0.4423
Epoch 5/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8460 - loss: 0.3718 - recall: 0.4097 - val_accuracy: 0.8440 - val_loss: 0.3656 - val_recall: 0.4423
Epoch 6/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8510 - loss: 0.3589 - recall: 0.4234 - val_accuracy: 0.8480 - val_loss: 0.3607 - val_recall: 0.4275
Epoch 7/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8560 - loss: 0.3512 - recall: 0.4387 - val_accuracy: 0.8505 - val_loss: 0.3585 - val_recall: 0.4423
Epoch 8/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8556 - loss: 0.3459 - recall: 0.4437 - val_accuracy: 0.8520 - val_loss: 0.3568 - val_recall: 0.4349
Epoch 9/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8566 - loss: 0.3411 - recall: 0.4554 - val_accuracy: 0.8520 - val_loss: 0.3561 - val_recall: 0.4201
Epoch 10/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8581 - loss: 0.3380 - recall: 0.4572 - val_accuracy: 0.8505 - val_loss: 0.3574 - val_recall: 0.4054
Epoch 11/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8580 - loss: 0.3357 - recall: 0.4631 - val_accuracy: 0.8500 - val_loss: 0.3578 - val_recall: 0.3956
Epoch 12/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8609 - loss: 0.3346 - recall: 0.4785 - val_accuracy: 0.8500 - val_loss: 0.3569 - val_recall: 0.4054
Epoch 13/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8620 - loss: 0.3321 - recall: 0.4814 - val_accuracy: 0.8530 - val_loss: 0.3562 - val_recall: 0.4029
Epoch 14/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8630 - loss: 0.3317 - recall: 0.4850 - val_accuracy: 0.8510 - val_loss: 0.3579 - val_recall: 0.4005
Epoch 15/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8637 - loss: 0.3308 - recall: 0.4866 - val_accuracy: 0.8510 - val_loss: 0.3579 - val_recall: 0.4029
Epoch 16/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8617 - loss: 0.3304 - recall: 0.4749 - val_accuracy: 0.8530 - val_loss: 0.3573 - val_recall: 0.4054
Epoch 17/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8620 - loss: 0.3291 - recall: 0.4767 - val_accuracy: 0.8495 - val_loss: 0.3589 - val_recall: 0.4029
Epoch 18/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8619 - loss: 0.3287 - recall: 0.4728 - val_accuracy: 0.8520 - val_loss: 0.3580 - val_recall: 0.4177
Epoch 19/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8604 - loss: 0.3277 - recall: 0.4717 - val_accuracy: 0.8530 - val_loss: 0.3601 - val_recall: 0.4152
Epoch 20/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8599 - loss: 0.3268 - recall: 0.4683 - val_accuracy: 0.8515 - val_loss: 0.3603 - val_recall: 0.4251
Epoch 21/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8603 - loss: 0.3261 - recall: 0.4708 - val_accuracy: 0.8520 - val_loss: 0.3600 - val_recall: 0.4300
Epoch 22/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8599 - loss: 0.3253 - recall: 0.4697 - val_accuracy: 0.8545 - val_loss: 0.3614 - val_recall: 0.4373
Epoch 23/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8591 - loss: 0.3245 - recall: 0.4706 - val_accuracy: 0.8530 - val_loss: 0.3619 - val_recall: 0.4447
Epoch 24/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8612 - loss: 0.3239 - recall: 0.4780 - val_accuracy: 0.8525 - val_loss: 0.3623 - val_recall: 0.4496
Epoch 25/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8609 - loss: 0.3236 - recall: 0.4752 - val_accuracy: 0.8520 - val_loss: 0.3625 - val_recall: 0.4521
Epoch 26/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8611 - loss: 0.3228 - recall: 0.4790 - val_accuracy: 0.8540 - val_loss: 0.3620 - val_recall: 0.4545
Epoch 27/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8615 - loss: 0.3217 - recall: 0.4757 - val_accuracy: 0.8540 - val_loss: 0.3632 - val_recall: 0.4521
Epoch 28/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8613 - loss: 0.3221 - recall: 0.4733 - val_accuracy: 0.8550 - val_loss: 0.3647 - val_recall: 0.4693
Epoch 29/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8625 - loss: 0.3209 - recall: 0.4811 - val_accuracy: 0.8545 - val_loss: 0.3664 - val_recall: 0.4619
Epoch 30/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.8621 - loss: 0.3207 - recall: 0.4800 - val_accuracy: 0.8530 - val_loss: 0.3665 - val_recall: 0.4619
Epoch 31/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8641 - loss: 0.3202 - recall: 0.4847 - val_accuracy: 0.8535 - val_loss: 0.3660 - val_recall: 0.4595
Epoch 32/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8634 - loss: 0.3193 - recall: 0.4811 - val_accuracy: 0.8550 - val_loss: 0.3659 - val_recall: 0.4570
Epoch 33/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8640 - loss: 0.3185 - recall: 0.4847 - val_accuracy: 0.8550 - val_loss: 0.3654 - val_recall: 0.4595
Epoch 34/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8644 - loss: 0.3182 - recall: 0.4897 - val_accuracy: 0.8545 - val_loss: 0.3646 - val_recall: 0.4619
Epoch 35/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8652 - loss: 0.3178 - recall: 0.4928 - val_accuracy: 0.8540 - val_loss: 0.3641 - val_recall: 0.4619
Epoch 36/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8662 - loss: 0.3171 - recall: 0.4968 - val_accuracy: 0.8525 - val_loss: 0.3683 - val_recall: 0.4570
Epoch 37/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8654 - loss: 0.3167 - recall: 0.4904 - val_accuracy: 0.8540 - val_loss: 0.3688 - val_recall: 0.4570
Epoch 38/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8655 - loss: 0.3165 - recall: 0.4919 - val_accuracy: 0.8505 - val_loss: 0.3687 - val_recall: 0.4472
Epoch 39/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8663 - loss: 0.3165 - recall: 0.4927 - val_accuracy: 0.8525 - val_loss: 0.3684 - val_recall: 0.4595
Epoch 40/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8674 - loss: 0.3157 - recall: 0.4922 - val_accuracy: 0.8495 - val_loss: 0.3694 - val_recall: 0.4496
In [ ]:
print("Time taken in seconds ",end-start)
Time taken in seconds  20.51025629043579
In [ ]:
plot(history,'loss')
In [ ]:
plot(history,'accuracy')
In [ ]:
plot(history,'recall')
In [ ]:
results.loc[1] = ['NN with Adam',2,'[64,32]','relu',epochs,batch_size,history.history["loss"][-1],history.history["val_loss"][-1],history.history["accuracy"][-1],history.history["val_accuracy"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results = results.sort_index()
results
Out[ ]:
model label # hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size train loss validation loss train accuracy validation accuracy train recall validation recall time (secs)
0 NN with SGD 2 [64,32] relu 100 64 0.33 0.35 0.86 0.86 0.47 0.43 42.23
1 NN with Adam 2 [64,32] relu 40 64 0.31 0.37 0.87 0.85 0.51 0.45 20.51

Commentary: This model achieved slightly better recall scores, while also not overfitting. The model also required fewer Epochs to converge, saving some compute time. So an improvement, but recall scores are still not where they need to be.

Neural Network with Adam Optimizer and Dropout¶

In [ ]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [ ]:
#Initializing the neural network
model_2 = Sequential()
model_2.add(Dense(64, activation="relu",input_dim = X_train.shape[1])) #Hidden & Input Layer
model_2.add(Dropout(0.3)) #Dropout Layer
model_2.add(Dense(32,activation="relu")) #Hidden Layer
model_2.add(Dropout(0.3)) #Dropout Layer
model_2.add(Dense(1,activation = "sigmoid")) #Output Layer
In [ ]:
#Printing the summary.
model_2.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             768 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout (Dropout)                    │ (None, 64)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_1 (Dropout)                  │ (None, 32)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
 Total params: 2,881 (11.25 KB)
 Trainable params: 2,881 (11.25 KB)
 Non-trainable params: 0 (0.00 B)
In [ ]:
optimizer = tf.keras.optimizers.Adam(learning_rate=0.005)    # defining Adam as the optimizer to be used
model_2.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy', tf.keras.metrics.Recall(name='recall')])
In [ ]:
#batch_size = X_train.shape[0]
batch_size = 64
epochs = 40

start = time.time()
history = model_2.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.7741 - loss: 0.5302 - recall: 0.0336 - val_accuracy: 0.7970 - val_loss: 0.4546 - val_recall: 0.0049
Epoch 2/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8043 - loss: 0.4548 - recall: 0.0663 - val_accuracy: 0.8220 - val_loss: 0.4307 - val_recall: 0.2285
Epoch 3/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8141 - loss: 0.4365 - recall: 0.1840 - val_accuracy: 0.8280 - val_loss: 0.4155 - val_recall: 0.2408
Epoch 4/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8203 - loss: 0.4178 - recall: 0.2533 - val_accuracy: 0.8385 - val_loss: 0.3989 - val_recall: 0.3243
Epoch 5/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8296 - loss: 0.4131 - recall: 0.2859 - val_accuracy: 0.8420 - val_loss: 0.3826 - val_recall: 0.3219
Epoch 6/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8377 - loss: 0.3972 - recall: 0.3312 - val_accuracy: 0.8495 - val_loss: 0.3688 - val_recall: 0.3563
Epoch 7/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8403 - loss: 0.3893 - recall: 0.3632 - val_accuracy: 0.8510 - val_loss: 0.3697 - val_recall: 0.3833
Epoch 8/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8476 - loss: 0.3737 - recall: 0.3608 - val_accuracy: 0.8530 - val_loss: 0.3660 - val_recall: 0.4005
Epoch 9/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8444 - loss: 0.3754 - recall: 0.3626 - val_accuracy: 0.8515 - val_loss: 0.3645 - val_recall: 0.3612
Epoch 10/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8548 - loss: 0.3675 - recall: 0.3818 - val_accuracy: 0.8485 - val_loss: 0.3627 - val_recall: 0.3956
Epoch 11/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8538 - loss: 0.3707 - recall: 0.3966 - val_accuracy: 0.8480 - val_loss: 0.3652 - val_recall: 0.4373
Epoch 12/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8515 - loss: 0.3659 - recall: 0.4077 - val_accuracy: 0.8510 - val_loss: 0.3619 - val_recall: 0.3710
Epoch 13/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8504 - loss: 0.3623 - recall: 0.3863 - val_accuracy: 0.8490 - val_loss: 0.3582 - val_recall: 0.3710
Epoch 14/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8516 - loss: 0.3636 - recall: 0.3820 - val_accuracy: 0.8505 - val_loss: 0.3608 - val_recall: 0.3882
Epoch 15/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8504 - loss: 0.3606 - recall: 0.3785 - val_accuracy: 0.8520 - val_loss: 0.3606 - val_recall: 0.3931
Epoch 16/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8539 - loss: 0.3617 - recall: 0.3904 - val_accuracy: 0.8470 - val_loss: 0.3611 - val_recall: 0.3464
Epoch 17/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8504 - loss: 0.3659 - recall: 0.3604 - val_accuracy: 0.8490 - val_loss: 0.3588 - val_recall: 0.3784
Epoch 18/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8501 - loss: 0.3607 - recall: 0.3784 - val_accuracy: 0.8500 - val_loss: 0.3584 - val_recall: 0.3857
Epoch 19/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step - accuracy: 0.8461 - loss: 0.3595 - recall: 0.3834 - val_accuracy: 0.8495 - val_loss: 0.3585 - val_recall: 0.3907
Epoch 20/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8553 - loss: 0.3633 - recall: 0.3904 - val_accuracy: 0.8500 - val_loss: 0.3580 - val_recall: 0.4029
Epoch 21/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8479 - loss: 0.3622 - recall: 0.3676 - val_accuracy: 0.8485 - val_loss: 0.3592 - val_recall: 0.3489
Epoch 22/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8532 - loss: 0.3598 - recall: 0.3902 - val_accuracy: 0.8530 - val_loss: 0.3600 - val_recall: 0.3857
Epoch 23/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 6ms/step - accuracy: 0.8544 - loss: 0.3556 - recall: 0.4016 - val_accuracy: 0.8510 - val_loss: 0.3559 - val_recall: 0.3784
Epoch 24/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8524 - loss: 0.3577 - recall: 0.3978 - val_accuracy: 0.8500 - val_loss: 0.3614 - val_recall: 0.3489
Epoch 25/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step - accuracy: 0.8563 - loss: 0.3489 - recall: 0.3972 - val_accuracy: 0.8525 - val_loss: 0.3579 - val_recall: 0.3538
Epoch 26/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8479 - loss: 0.3583 - recall: 0.3778 - val_accuracy: 0.8525 - val_loss: 0.3575 - val_recall: 0.3808
Epoch 27/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8534 - loss: 0.3586 - recall: 0.3929 - val_accuracy: 0.8515 - val_loss: 0.3584 - val_recall: 0.4177
Epoch 28/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8566 - loss: 0.3535 - recall: 0.4347 - val_accuracy: 0.8525 - val_loss: 0.3606 - val_recall: 0.4128
Epoch 29/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8532 - loss: 0.3576 - recall: 0.4226 - val_accuracy: 0.8510 - val_loss: 0.3570 - val_recall: 0.4029
Epoch 30/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8530 - loss: 0.3552 - recall: 0.4385 - val_accuracy: 0.8510 - val_loss: 0.3585 - val_recall: 0.3735
Epoch 31/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8544 - loss: 0.3521 - recall: 0.3942 - val_accuracy: 0.8560 - val_loss: 0.3545 - val_recall: 0.4423
Epoch 32/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8553 - loss: 0.3539 - recall: 0.4226 - val_accuracy: 0.8540 - val_loss: 0.3567 - val_recall: 0.4103
Epoch 33/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8557 - loss: 0.3502 - recall: 0.4299 - val_accuracy: 0.8525 - val_loss: 0.3605 - val_recall: 0.4079
Epoch 34/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8564 - loss: 0.3561 - recall: 0.4146 - val_accuracy: 0.8530 - val_loss: 0.3626 - val_recall: 0.4324
Epoch 35/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8565 - loss: 0.3507 - recall: 0.4456 - val_accuracy: 0.8530 - val_loss: 0.3572 - val_recall: 0.4029
Epoch 36/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - accuracy: 0.8565 - loss: 0.3546 - recall: 0.4068 - val_accuracy: 0.8580 - val_loss: 0.3541 - val_recall: 0.4152
Epoch 37/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8550 - loss: 0.3485 - recall: 0.4025 - val_accuracy: 0.8570 - val_loss: 0.3562 - val_recall: 0.4373
Epoch 38/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8580 - loss: 0.3464 - recall: 0.4223 - val_accuracy: 0.8530 - val_loss: 0.3573 - val_recall: 0.3931
Epoch 39/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8563 - loss: 0.3510 - recall: 0.4194 - val_accuracy: 0.8475 - val_loss: 0.3650 - val_recall: 0.3514
Epoch 40/40
94/94 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8556 - loss: 0.3553 - recall: 0.4090 - val_accuracy: 0.8510 - val_loss: 0.3564 - val_recall: 0.4177
In [ ]:
print("Time taken in seconds ",end-start)
Time taken in seconds  20.626485586166382
In [ ]:
plot(history,'loss')
In [ ]:
plot(history,'accuracy')
In [ ]:
plot(history,'recall')
In [ ]:
results.loc[2] = ['NN with Adam/Dropout',2,'[64,32]','relu',epochs,batch_size,history.history["loss"][-1],history.history["val_loss"][-1],history.history["accuracy"][-1],history.history["val_accuracy"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results = results.sort_index()
results
Out[ ]:
model label # hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size train loss validation loss train accuracy validation accuracy train recall validation recall time (secs)
0 NN with SGD 2 [64,32] relu 100 64 0.33 0.35 0.86 0.86 0.47 0.43 42.23
1 NN with Adam 2 [64,32] relu 40 64 0.31 0.37 0.87 0.85 0.51 0.45 20.51
2 NN with Adam/Dropout 2 [64,32] relu 40 64 0.35 0.36 0.86 0.85 0.43 0.42 20.63

Commentary: This model reduced the already modest overfitting from the previous two models, however, the recall scores are worse, so all-in-all, not an improvement with this one.

Neural Network with Balanced Data (by applying SMOTE) and SGD Optimizer¶

In [ ]:
#Applying SMOTE to handle class imbalance
X_train, y_train = smote.fit_resample(X_train, y_train)
In [ ]:
# Print the updated shape of the training data
print("Training set shape:", X_train.shape, y_train.shape)
Training set shape: (9554, 11) (9554,)
In [ ]:
#Show the updated balance of y_train
y_train.value_counts()
Out[ ]:
count
Exited
0 4777
1 4777

In [ ]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [ ]:
#Initializing the neural network
model_3 = Sequential()
model_3.add(Dense(64, activation="relu",input_dim = X_train.shape[1]))
model_3.add(Dense(32,activation="relu"))
model_3.add(Dense(1,activation = "sigmoid"))
In [ ]:
#Printing the summary.
model_3.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             768 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
 Total params: 2,881 (11.25 KB)
 Trainable params: 2,881 (11.25 KB)
 Non-trainable params: 0 (0.00 B)
In [ ]:
optimizer = keras.optimizers.SGD(learning_rate=0.005, momentum=0.9)    # defining SGD as the optimizer to be used
model_3.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy', tf.keras.metrics.Recall(name='recall')])
In [ ]:
#batch_size = X_train.shape[0]
batch_size = 64
epochs = 100

start = time.time()
history = model_3.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 4ms/step - accuracy: 0.5156 - loss: 0.6937 - recall: 0.6299 - val_accuracy: 0.5530 - val_loss: 0.6862 - val_recall: 0.7224
Epoch 2/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.6465 - loss: 0.6476 - recall: 0.6828 - val_accuracy: 0.5800 - val_loss: 0.6772 - val_recall: 0.7027
Epoch 3/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.6635 - loss: 0.6237 - recall: 0.6784 - val_accuracy: 0.5965 - val_loss: 0.6660 - val_recall: 0.7052
Epoch 4/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.6698 - loss: 0.6098 - recall: 0.6803 - val_accuracy: 0.6180 - val_loss: 0.6510 - val_recall: 0.7076
Epoch 5/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.6854 - loss: 0.5975 - recall: 0.6926 - val_accuracy: 0.6460 - val_loss: 0.6277 - val_recall: 0.7150
Epoch 6/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7020 - loss: 0.5859 - recall: 0.7083 - val_accuracy: 0.6790 - val_loss: 0.6034 - val_recall: 0.6929
Epoch 7/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7062 - loss: 0.5760 - recall: 0.7112 - val_accuracy: 0.6940 - val_loss: 0.5919 - val_recall: 0.7027
Epoch 8/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7124 - loss: 0.5687 - recall: 0.7135 - val_accuracy: 0.7000 - val_loss: 0.5862 - val_recall: 0.7076
Epoch 9/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7144 - loss: 0.5632 - recall: 0.7114 - val_accuracy: 0.7060 - val_loss: 0.5833 - val_recall: 0.7174
Epoch 10/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7190 - loss: 0.5583 - recall: 0.7123 - val_accuracy: 0.7100 - val_loss: 0.5816 - val_recall: 0.7224
Epoch 11/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7245 - loss: 0.5536 - recall: 0.7193 - val_accuracy: 0.7145 - val_loss: 0.5804 - val_recall: 0.7273
Epoch 12/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7289 - loss: 0.5486 - recall: 0.7221 - val_accuracy: 0.7175 - val_loss: 0.5776 - val_recall: 0.7322
Epoch 13/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7351 - loss: 0.5427 - recall: 0.7332 - val_accuracy: 0.7175 - val_loss: 0.5743 - val_recall: 0.7322
Epoch 14/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7405 - loss: 0.5356 - recall: 0.7379 - val_accuracy: 0.7245 - val_loss: 0.5708 - val_recall: 0.7469
Epoch 15/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7480 - loss: 0.5268 - recall: 0.7428 - val_accuracy: 0.7245 - val_loss: 0.5732 - val_recall: 0.7666
Epoch 16/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7545 - loss: 0.5174 - recall: 0.7504 - val_accuracy: 0.7115 - val_loss: 0.5829 - val_recall: 0.7838
Epoch 17/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7577 - loss: 0.5088 - recall: 0.7571 - val_accuracy: 0.7045 - val_loss: 0.5907 - val_recall: 0.7936
Epoch 18/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7647 - loss: 0.5003 - recall: 0.7642 - val_accuracy: 0.6880 - val_loss: 0.6013 - val_recall: 0.8059
Epoch 19/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7647 - loss: 0.4933 - recall: 0.7646 - val_accuracy: 0.6750 - val_loss: 0.6121 - val_recall: 0.8182
Epoch 20/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7671 - loss: 0.4875 - recall: 0.7641 - val_accuracy: 0.6685 - val_loss: 0.6222 - val_recall: 0.8378
Epoch 21/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7706 - loss: 0.4827 - recall: 0.7690 - val_accuracy: 0.6630 - val_loss: 0.6284 - val_recall: 0.8452
Epoch 22/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7750 - loss: 0.4786 - recall: 0.7744 - val_accuracy: 0.6620 - val_loss: 0.6282 - val_recall: 0.8501
Epoch 23/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7741 - loss: 0.4748 - recall: 0.7697 - val_accuracy: 0.6635 - val_loss: 0.6335 - val_recall: 0.8550
Epoch 24/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7757 - loss: 0.4715 - recall: 0.7684 - val_accuracy: 0.6600 - val_loss: 0.6364 - val_recall: 0.8526
Epoch 25/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7764 - loss: 0.4690 - recall: 0.7680 - val_accuracy: 0.6570 - val_loss: 0.6409 - val_recall: 0.8673
Epoch 26/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7752 - loss: 0.4672 - recall: 0.7663 - val_accuracy: 0.6530 - val_loss: 0.6463 - val_recall: 0.8722
Epoch 27/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7753 - loss: 0.4657 - recall: 0.7650 - val_accuracy: 0.6485 - val_loss: 0.6488 - val_recall: 0.8747
Epoch 28/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7783 - loss: 0.4639 - recall: 0.7670 - val_accuracy: 0.6465 - val_loss: 0.6502 - val_recall: 0.8771
Epoch 29/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7794 - loss: 0.4623 - recall: 0.7692 - val_accuracy: 0.6510 - val_loss: 0.6423 - val_recall: 0.8722
Epoch 30/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7805 - loss: 0.4608 - recall: 0.7695 - val_accuracy: 0.6535 - val_loss: 0.6378 - val_recall: 0.8722
Epoch 31/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7807 - loss: 0.4594 - recall: 0.7683 - val_accuracy: 0.6550 - val_loss: 0.6362 - val_recall: 0.8747
Epoch 32/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7810 - loss: 0.4581 - recall: 0.7686 - val_accuracy: 0.6530 - val_loss: 0.6364 - val_recall: 0.8771
Epoch 33/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7837 - loss: 0.4571 - recall: 0.7724 - val_accuracy: 0.6540 - val_loss: 0.6361 - val_recall: 0.8722
Epoch 34/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7854 - loss: 0.4555 - recall: 0.7730 - val_accuracy: 0.6605 - val_loss: 0.6297 - val_recall: 0.8649
Epoch 35/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7856 - loss: 0.4553 - recall: 0.7728 - val_accuracy: 0.6615 - val_loss: 0.6245 - val_recall: 0.8649
Epoch 36/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7866 - loss: 0.4537 - recall: 0.7737 - val_accuracy: 0.6650 - val_loss: 0.6229 - val_recall: 0.8600
Epoch 37/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7863 - loss: 0.4530 - recall: 0.7727 - val_accuracy: 0.6665 - val_loss: 0.6233 - val_recall: 0.8600
Epoch 38/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7867 - loss: 0.4523 - recall: 0.7738 - val_accuracy: 0.6695 - val_loss: 0.6146 - val_recall: 0.8550
Epoch 39/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7879 - loss: 0.4511 - recall: 0.7740 - val_accuracy: 0.6660 - val_loss: 0.6223 - val_recall: 0.8649
Epoch 40/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7884 - loss: 0.4509 - recall: 0.7749 - val_accuracy: 0.6660 - val_loss: 0.6200 - val_recall: 0.8575
Epoch 41/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7889 - loss: 0.4501 - recall: 0.7748 - val_accuracy: 0.6685 - val_loss: 0.6180 - val_recall: 0.8600
Epoch 42/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7900 - loss: 0.4490 - recall: 0.7758 - val_accuracy: 0.6675 - val_loss: 0.6205 - val_recall: 0.8624
Epoch 43/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7891 - loss: 0.4492 - recall: 0.7773 - val_accuracy: 0.6675 - val_loss: 0.6213 - val_recall: 0.8575
Epoch 44/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7890 - loss: 0.4476 - recall: 0.7777 - val_accuracy: 0.6700 - val_loss: 0.6169 - val_recall: 0.8550
Epoch 45/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7899 - loss: 0.4473 - recall: 0.7798 - val_accuracy: 0.6720 - val_loss: 0.6153 - val_recall: 0.8575
Epoch 46/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7911 - loss: 0.4463 - recall: 0.7793 - val_accuracy: 0.6730 - val_loss: 0.6170 - val_recall: 0.8575
Epoch 47/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7921 - loss: 0.4454 - recall: 0.7806 - val_accuracy: 0.6810 - val_loss: 0.6038 - val_recall: 0.8501
Epoch 48/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.7937 - loss: 0.4436 - recall: 0.7817 - val_accuracy: 0.6825 - val_loss: 0.5997 - val_recall: 0.8452
Epoch 49/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7934 - loss: 0.4434 - recall: 0.7803 - val_accuracy: 0.6810 - val_loss: 0.6037 - val_recall: 0.8452
Epoch 50/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7920 - loss: 0.4432 - recall: 0.7795 - val_accuracy: 0.6805 - val_loss: 0.6032 - val_recall: 0.8452
Epoch 51/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7940 - loss: 0.4421 - recall: 0.7806 - val_accuracy: 0.6855 - val_loss: 0.5988 - val_recall: 0.8428
Epoch 52/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7935 - loss: 0.4412 - recall: 0.7796 - val_accuracy: 0.6830 - val_loss: 0.6002 - val_recall: 0.8403
Epoch 53/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7939 - loss: 0.4406 - recall: 0.7813 - val_accuracy: 0.6855 - val_loss: 0.5993 - val_recall: 0.8403
Epoch 54/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7951 - loss: 0.4395 - recall: 0.7824 - val_accuracy: 0.6830 - val_loss: 0.6013 - val_recall: 0.8403
Epoch 55/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7961 - loss: 0.4389 - recall: 0.7844 - val_accuracy: 0.6820 - val_loss: 0.6032 - val_recall: 0.8452
Epoch 56/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7963 - loss: 0.4384 - recall: 0.7845 - val_accuracy: 0.6820 - val_loss: 0.6031 - val_recall: 0.8378
Epoch 57/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7973 - loss: 0.4380 - recall: 0.7849 - val_accuracy: 0.6875 - val_loss: 0.5954 - val_recall: 0.8354
Epoch 58/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7983 - loss: 0.4366 - recall: 0.7865 - val_accuracy: 0.6860 - val_loss: 0.5956 - val_recall: 0.8305
Epoch 59/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7974 - loss: 0.4365 - recall: 0.7861 - val_accuracy: 0.6860 - val_loss: 0.5938 - val_recall: 0.8305
Epoch 60/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8005 - loss: 0.4353 - recall: 0.7907 - val_accuracy: 0.6895 - val_loss: 0.5910 - val_recall: 0.8305
Epoch 61/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8001 - loss: 0.4346 - recall: 0.7899 - val_accuracy: 0.6870 - val_loss: 0.5945 - val_recall: 0.8329
Epoch 62/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8014 - loss: 0.4338 - recall: 0.7913 - val_accuracy: 0.6890 - val_loss: 0.5931 - val_recall: 0.8305
Epoch 63/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8014 - loss: 0.4331 - recall: 0.7918 - val_accuracy: 0.6880 - val_loss: 0.5916 - val_recall: 0.8256
Epoch 64/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8007 - loss: 0.4330 - recall: 0.7903 - val_accuracy: 0.6860 - val_loss: 0.6037 - val_recall: 0.8403
Epoch 65/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8021 - loss: 0.4323 - recall: 0.7931 - val_accuracy: 0.6880 - val_loss: 0.5997 - val_recall: 0.8403
Epoch 66/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8022 - loss: 0.4315 - recall: 0.7926 - val_accuracy: 0.6855 - val_loss: 0.6061 - val_recall: 0.8452
Epoch 67/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8029 - loss: 0.4315 - recall: 0.7935 - val_accuracy: 0.6865 - val_loss: 0.6048 - val_recall: 0.8428
Epoch 68/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8023 - loss: 0.4306 - recall: 0.7931 - val_accuracy: 0.6870 - val_loss: 0.6026 - val_recall: 0.8403
Epoch 69/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8035 - loss: 0.4294 - recall: 0.7938 - val_accuracy: 0.6940 - val_loss: 0.5890 - val_recall: 0.8280
Epoch 70/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8028 - loss: 0.4294 - recall: 0.7927 - val_accuracy: 0.6850 - val_loss: 0.6072 - val_recall: 0.8428
Epoch 71/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8029 - loss: 0.4288 - recall: 0.7948 - val_accuracy: 0.6940 - val_loss: 0.5909 - val_recall: 0.8329
Epoch 72/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8047 - loss: 0.4275 - recall: 0.7948 - val_accuracy: 0.6950 - val_loss: 0.5904 - val_recall: 0.8329
Epoch 73/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8045 - loss: 0.4272 - recall: 0.7946 - val_accuracy: 0.6985 - val_loss: 0.5860 - val_recall: 0.8305
Epoch 74/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8054 - loss: 0.4266 - recall: 0.7952 - val_accuracy: 0.6990 - val_loss: 0.5833 - val_recall: 0.8280
Epoch 75/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8065 - loss: 0.4264 - recall: 0.7968 - val_accuracy: 0.6875 - val_loss: 0.6035 - val_recall: 0.8378
Epoch 76/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8038 - loss: 0.4266 - recall: 0.7955 - val_accuracy: 0.6855 - val_loss: 0.6072 - val_recall: 0.8403
Epoch 77/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8051 - loss: 0.4259 - recall: 0.7979 - val_accuracy: 0.6835 - val_loss: 0.6103 - val_recall: 0.8428
Epoch 78/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8056 - loss: 0.4254 - recall: 0.7970 - val_accuracy: 0.6835 - val_loss: 0.6092 - val_recall: 0.8428
Epoch 79/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8053 - loss: 0.4250 - recall: 0.7975 - val_accuracy: 0.6825 - val_loss: 0.6082 - val_recall: 0.8403
Epoch 80/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8064 - loss: 0.4236 - recall: 0.8002 - val_accuracy: 0.6895 - val_loss: 0.6008 - val_recall: 0.8354
Epoch 81/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8076 - loss: 0.4229 - recall: 0.8010 - val_accuracy: 0.6840 - val_loss: 0.6048 - val_recall: 0.8378
Epoch 82/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8064 - loss: 0.4221 - recall: 0.8003 - val_accuracy: 0.6850 - val_loss: 0.6052 - val_recall: 0.8378
Epoch 83/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8081 - loss: 0.4216 - recall: 0.8025 - val_accuracy: 0.6890 - val_loss: 0.6043 - val_recall: 0.8403
Epoch 84/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8085 - loss: 0.4214 - recall: 0.8016 - val_accuracy: 0.6845 - val_loss: 0.6068 - val_recall: 0.8403
Epoch 85/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8089 - loss: 0.4207 - recall: 0.8022 - val_accuracy: 0.6825 - val_loss: 0.6134 - val_recall: 0.8452
Epoch 86/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8082 - loss: 0.4196 - recall: 0.8029 - val_accuracy: 0.6850 - val_loss: 0.6113 - val_recall: 0.8452
Epoch 87/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8093 - loss: 0.4191 - recall: 0.8023 - val_accuracy: 0.6835 - val_loss: 0.6152 - val_recall: 0.8477
Epoch 88/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8091 - loss: 0.4182 - recall: 0.8035 - val_accuracy: 0.6840 - val_loss: 0.6166 - val_recall: 0.8477
Epoch 89/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8095 - loss: 0.4171 - recall: 0.8041 - val_accuracy: 0.6865 - val_loss: 0.6119 - val_recall: 0.8501
Epoch 90/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8089 - loss: 0.4167 - recall: 0.8021 - val_accuracy: 0.6865 - val_loss: 0.6116 - val_recall: 0.8501
Epoch 91/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8100 - loss: 0.4165 - recall: 0.8038 - val_accuracy: 0.6875 - val_loss: 0.6122 - val_recall: 0.8452
Epoch 92/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8088 - loss: 0.4148 - recall: 0.8039 - val_accuracy: 0.6930 - val_loss: 0.6040 - val_recall: 0.8403
Epoch 93/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8103 - loss: 0.4137 - recall: 0.8046 - val_accuracy: 0.6940 - val_loss: 0.6058 - val_recall: 0.8428
Epoch 94/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8115 - loss: 0.4134 - recall: 0.8048 - val_accuracy: 0.6915 - val_loss: 0.6087 - val_recall: 0.8428
Epoch 95/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8111 - loss: 0.4124 - recall: 0.8050 - val_accuracy: 0.6925 - val_loss: 0.6059 - val_recall: 0.8403
Epoch 96/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8110 - loss: 0.4124 - recall: 0.8058 - val_accuracy: 0.6980 - val_loss: 0.5975 - val_recall: 0.8378
Epoch 97/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8126 - loss: 0.4113 - recall: 0.8065 - val_accuracy: 0.6925 - val_loss: 0.6073 - val_recall: 0.8403
Epoch 98/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8126 - loss: 0.4106 - recall: 0.8067 - val_accuracy: 0.6920 - val_loss: 0.6094 - val_recall: 0.8428
Epoch 99/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8140 - loss: 0.4103 - recall: 0.8095 - val_accuracy: 0.6925 - val_loss: 0.6094 - val_recall: 0.8428
Epoch 100/100
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8132 - loss: 0.4094 - recall: 0.8090 - val_accuracy: 0.6970 - val_loss: 0.6040 - val_recall: 0.8403
In [ ]:
print("Time taken in seconds ",end-start)
Time taken in seconds  68.57739901542664
In [ ]:
plot(history,'loss')
In [ ]:
plot(history,'accuracy')
In [ ]:
plot(history,'recall')
#print(history.history.keys())
In [ ]:
results.loc[3] = ['NN SMOTE with SGD',2,'[64,32]','relu',epochs,batch_size,history.history["loss"][-1],history.history["val_loss"][-1],history.history["accuracy"][-1],history.history["val_accuracy"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results = results.sort_index()
results
Out[ ]:
model label # hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size train loss validation loss train accuracy validation accuracy train recall validation recall time (secs)
0 NN with SGD 2 [64,32] relu 100 64 0.33 0.35 0.86 0.86 0.47 0.43 42.23
1 NN with Adam 2 [64,32] relu 40 64 0.31 0.37 0.87 0.85 0.51 0.45 20.51
2 NN with Adam/Dropout 2 [64,32] relu 40 64 0.35 0.36 0.86 0.85 0.43 0.42 20.63
3 NN SMOTE with SGD 2 [64,32] relu 100 64 0.40 0.60 0.82 0.70 0.82 0.84 68.58

Commentary: This model has achieved much better recall, but the diverging behavior of accuracy for train and validation suggests significant overfitting.

Neural Network with Balanced Data (by applying SMOTE) and Adam Optimizer¶

In [ ]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [ ]:
#Initializing the neural network
model_4 = Sequential()
model_4.add(Dense(64, activation="relu",input_dim = X_train.shape[1]))
model_4.add(Dense(32,activation="relu"))
model_4.add(Dense(1,activation = "sigmoid"))
In [ ]:
#Printing the summary.
model_4.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             768 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
 Total params: 2,881 (11.25 KB)
 Trainable params: 2,881 (11.25 KB)
 Non-trainable params: 0 (0.00 B)
In [ ]:
optimizer = tf.keras.optimizers.Adam(learning_rate=0.005)    # defining Adam as the optimizer to be used
model_4.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy', tf.keras.metrics.Recall(name='recall')])
In [ ]:
#batch_size = X_train.shape[0]
batch_size = 64
epochs = 40

start = time.time()
history = model_4.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.6459 - loss: 0.6217 - recall: 0.6060 - val_accuracy: 0.6305 - val_loss: 0.6795 - val_recall: 0.8501
Epoch 2/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7486 - loss: 0.5154 - recall: 0.7435 - val_accuracy: 0.6580 - val_loss: 0.6305 - val_recall: 0.8477
Epoch 3/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7703 - loss: 0.4804 - recall: 0.7630 - val_accuracy: 0.6625 - val_loss: 0.6034 - val_recall: 0.8354
Epoch 4/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7767 - loss: 0.4649 - recall: 0.7681 - val_accuracy: 0.6725 - val_loss: 0.5920 - val_recall: 0.8403
Epoch 5/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7817 - loss: 0.4542 - recall: 0.7775 - val_accuracy: 0.6870 - val_loss: 0.5781 - val_recall: 0.8182
Epoch 6/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7907 - loss: 0.4416 - recall: 0.7859 - val_accuracy: 0.6970 - val_loss: 0.5704 - val_recall: 0.8034
Epoch 7/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7950 - loss: 0.4349 - recall: 0.7946 - val_accuracy: 0.7125 - val_loss: 0.5507 - val_recall: 0.7912
Epoch 8/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.7979 - loss: 0.4303 - recall: 0.7980 - val_accuracy: 0.7185 - val_loss: 0.5526 - val_recall: 0.7862
Epoch 9/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7987 - loss: 0.4261 - recall: 0.7990 - val_accuracy: 0.7215 - val_loss: 0.5517 - val_recall: 0.7789
Epoch 10/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8042 - loss: 0.4217 - recall: 0.8048 - val_accuracy: 0.7245 - val_loss: 0.5524 - val_recall: 0.7862
Epoch 11/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8045 - loss: 0.4170 - recall: 0.8038 - val_accuracy: 0.7280 - val_loss: 0.5481 - val_recall: 0.7862
Epoch 12/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8072 - loss: 0.4129 - recall: 0.8079 - val_accuracy: 0.7285 - val_loss: 0.5467 - val_recall: 0.7715
Epoch 13/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8060 - loss: 0.4088 - recall: 0.8110 - val_accuracy: 0.7370 - val_loss: 0.5351 - val_recall: 0.7690
Epoch 14/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8086 - loss: 0.4032 - recall: 0.8154 - val_accuracy: 0.7585 - val_loss: 0.5071 - val_recall: 0.7273
Epoch 15/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8125 - loss: 0.3990 - recall: 0.8190 - val_accuracy: 0.7580 - val_loss: 0.5134 - val_recall: 0.7543
Epoch 16/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8139 - loss: 0.3957 - recall: 0.8218 - val_accuracy: 0.7435 - val_loss: 0.5393 - val_recall: 0.7789
Epoch 17/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8151 - loss: 0.3928 - recall: 0.8249 - val_accuracy: 0.7585 - val_loss: 0.5199 - val_recall: 0.7543
Epoch 18/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - accuracy: 0.8200 - loss: 0.3888 - recall: 0.8320 - val_accuracy: 0.7435 - val_loss: 0.5543 - val_recall: 0.7838
Epoch 19/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8210 - loss: 0.3865 - recall: 0.8340 - val_accuracy: 0.7490 - val_loss: 0.5410 - val_recall: 0.7617
Epoch 20/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8213 - loss: 0.3833 - recall: 0.8359 - val_accuracy: 0.7520 - val_loss: 0.5399 - val_recall: 0.7666
Epoch 21/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8211 - loss: 0.3787 - recall: 0.8334 - val_accuracy: 0.7450 - val_loss: 0.5454 - val_recall: 0.7469
Epoch 22/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8251 - loss: 0.3777 - recall: 0.8399 - val_accuracy: 0.7615 - val_loss: 0.5235 - val_recall: 0.7224
Epoch 23/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8271 - loss: 0.3736 - recall: 0.8408 - val_accuracy: 0.7725 - val_loss: 0.5104 - val_recall: 0.7101
Epoch 24/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8303 - loss: 0.3686 - recall: 0.8404 - val_accuracy: 0.7530 - val_loss: 0.5420 - val_recall: 0.7420
Epoch 25/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8317 - loss: 0.3646 - recall: 0.8470 - val_accuracy: 0.7665 - val_loss: 0.5238 - val_recall: 0.7248
Epoch 26/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8345 - loss: 0.3614 - recall: 0.8493 - val_accuracy: 0.7645 - val_loss: 0.5296 - val_recall: 0.7199
Epoch 27/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8341 - loss: 0.3613 - recall: 0.8470 - val_accuracy: 0.7665 - val_loss: 0.5282 - val_recall: 0.7199
Epoch 28/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8403 - loss: 0.3566 - recall: 0.8568 - val_accuracy: 0.7660 - val_loss: 0.5293 - val_recall: 0.7150
Epoch 29/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - accuracy: 0.8389 - loss: 0.3533 - recall: 0.8557 - val_accuracy: 0.7555 - val_loss: 0.5445 - val_recall: 0.7346
Epoch 30/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8390 - loss: 0.3519 - recall: 0.8568 - val_accuracy: 0.7595 - val_loss: 0.5468 - val_recall: 0.7346
Epoch 31/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8417 - loss: 0.3501 - recall: 0.8579 - val_accuracy: 0.7615 - val_loss: 0.5425 - val_recall: 0.7371
Epoch 32/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8419 - loss: 0.3470 - recall: 0.8623 - val_accuracy: 0.7630 - val_loss: 0.5478 - val_recall: 0.7396
Epoch 33/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8441 - loss: 0.3475 - recall: 0.8651 - val_accuracy: 0.7585 - val_loss: 0.5490 - val_recall: 0.7469
Epoch 34/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8431 - loss: 0.3425 - recall: 0.8627 - val_accuracy: 0.7635 - val_loss: 0.5434 - val_recall: 0.7322
Epoch 35/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 6ms/step - accuracy: 0.8490 - loss: 0.3396 - recall: 0.8671 - val_accuracy: 0.7535 - val_loss: 0.5625 - val_recall: 0.7469
Epoch 36/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8479 - loss: 0.3386 - recall: 0.8686 - val_accuracy: 0.7645 - val_loss: 0.5476 - val_recall: 0.7518
Epoch 37/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8505 - loss: 0.3344 - recall: 0.8705 - val_accuracy: 0.7670 - val_loss: 0.5402 - val_recall: 0.7371
Epoch 38/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8511 - loss: 0.3308 - recall: 0.8726 - val_accuracy: 0.7755 - val_loss: 0.5329 - val_recall: 0.7322
Epoch 39/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8519 - loss: 0.3297 - recall: 0.8754 - val_accuracy: 0.7675 - val_loss: 0.5456 - val_recall: 0.7445
Epoch 40/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8525 - loss: 0.3261 - recall: 0.8765 - val_accuracy: 0.7640 - val_loss: 0.5571 - val_recall: 0.7420
In [ ]:
print("Time taken in seconds ",end-start)
Time taken in seconds  29.962656497955322
In [ ]:
plot(history,'loss')
In [ ]:
plot(history,'accuracy')
In [ ]:
plot(history,'recall')
In [ ]:
results.loc[4] = ['NN SMOTE with Adam',2,'[64,32]','relu',epochs,batch_size,history.history["loss"][-1],history.history["val_loss"][-1],history.history["accuracy"][-1],history.history["val_accuracy"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results = results.sort_index()
results
Out[ ]:
model label # hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size train loss validation loss train accuracy validation accuracy train recall validation recall time (secs)
0 NN with SGD 2 [64,32] relu 100 64 0.33 0.35 0.86 0.86 0.47 0.43 42.23
1 NN with Adam 2 [64,32] relu 40 64 0.31 0.37 0.87 0.85 0.51 0.45 20.51
2 NN with Adam/Dropout 2 [64,32] relu 40 64 0.35 0.36 0.86 0.85 0.43 0.42 20.63
3 NN SMOTE with SGD 2 [64,32] relu 100 64 0.40 0.60 0.82 0.70 0.82 0.84 68.58
4 NN SMOTE with Adam 2 [64,32] relu 40 64 0.32 0.56 0.86 0.76 0.88 0.74 29.96

Commentary: This model has very high recall for Train, but the Validation Metrics are much worse indicating significant overfitting.

Neural Network with Balanced Data (by applying SMOTE), Adam Optimizer, and Dropout¶

In [ ]:
# clears the current Keras session, resetting all layers and models previously created, freeing up memory and resources.
tf.keras.backend.clear_session()
In [ ]:
#Initializing the neural network
model_5 = Sequential()
model_5.add(Dense(64, activation="relu",input_dim = X_train.shape[1])) #Input Layer
model_5.add(Dropout(0.3)) #Dropout Layer
model_5.add(Dense(32,activation="relu")) #Hidden Layer
model_5.add(Dropout(0.3)) #Dropout Layer
model_5.add(Dense(1,activation = "sigmoid")) #Output Layer
In [ ]:
#Printing the summary.
model_5.summary()
Model: "sequential"
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━┓
┃ Layer (type)                         ┃ Output Shape                ┃         Param # ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━┩
│ dense (Dense)                        │ (None, 64)                  │             768 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout (Dropout)                    │ (None, 64)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_1 (Dense)                      │ (None, 32)                  │           2,080 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dropout_1 (Dropout)                  │ (None, 32)                  │               0 │
├──────────────────────────────────────┼─────────────────────────────┼─────────────────┤
│ dense_2 (Dense)                      │ (None, 1)                   │              33 │
└──────────────────────────────────────┴─────────────────────────────┴─────────────────┘
 Total params: 2,881 (11.25 KB)
 Trainable params: 2,881 (11.25 KB)
 Non-trainable params: 0 (0.00 B)
In [ ]:
optimizer = tf.keras.optimizers.Adam(learning_rate=0.005)    # defining Adam as the optimizer to be used
model_5.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy', tf.keras.metrics.Recall(name='recall')])
In [ ]:
batch_size = 64
epochs = 40

start = time.time()
history = model_5.fit(X_train, y_train, validation_data=(X_val,y_val) , batch_size=batch_size, epochs=epochs)
end=time.time()
Epoch 1/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 2s 5ms/step - accuracy: 0.6053 - loss: 0.6538 - recall: 0.6135 - val_accuracy: 0.6040 - val_loss: 0.6819 - val_recall: 0.8378
Epoch 2/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7055 - loss: 0.5701 - recall: 0.7161 - val_accuracy: 0.6085 - val_loss: 0.6689 - val_recall: 0.8870
Epoch 3/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7414 - loss: 0.5237 - recall: 0.7434 - val_accuracy: 0.7030 - val_loss: 0.5422 - val_recall: 0.8133
Epoch 4/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7571 - loss: 0.5053 - recall: 0.7583 - val_accuracy: 0.6955 - val_loss: 0.5485 - val_recall: 0.8256
Epoch 5/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7654 - loss: 0.4868 - recall: 0.7723 - val_accuracy: 0.7110 - val_loss: 0.5499 - val_recall: 0.8305
Epoch 6/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7658 - loss: 0.4834 - recall: 0.7653 - val_accuracy: 0.7045 - val_loss: 0.5381 - val_recall: 0.8354
Epoch 7/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7719 - loss: 0.4787 - recall: 0.7772 - val_accuracy: 0.7280 - val_loss: 0.5118 - val_recall: 0.7862
Epoch 8/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7755 - loss: 0.4688 - recall: 0.7757 - val_accuracy: 0.6945 - val_loss: 0.5403 - val_recall: 0.8378
Epoch 9/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7740 - loss: 0.4730 - recall: 0.7926 - val_accuracy: 0.7265 - val_loss: 0.5138 - val_recall: 0.8034
Epoch 10/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7728 - loss: 0.4688 - recall: 0.7706 - val_accuracy: 0.7440 - val_loss: 0.4987 - val_recall: 0.7666
Epoch 11/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7858 - loss: 0.4571 - recall: 0.7921 - val_accuracy: 0.7420 - val_loss: 0.5067 - val_recall: 0.7568
Epoch 12/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7835 - loss: 0.4579 - recall: 0.7870 - val_accuracy: 0.7505 - val_loss: 0.4939 - val_recall: 0.7715
Epoch 13/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7862 - loss: 0.4554 - recall: 0.7910 - val_accuracy: 0.7485 - val_loss: 0.4977 - val_recall: 0.7543
Epoch 14/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7823 - loss: 0.4595 - recall: 0.7861 - val_accuracy: 0.7685 - val_loss: 0.4756 - val_recall: 0.7346
Epoch 15/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7894 - loss: 0.4470 - recall: 0.7883 - val_accuracy: 0.7735 - val_loss: 0.4682 - val_recall: 0.7297
Epoch 16/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7880 - loss: 0.4554 - recall: 0.7914 - val_accuracy: 0.7520 - val_loss: 0.4939 - val_recall: 0.7592
Epoch 17/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7861 - loss: 0.4491 - recall: 0.7905 - val_accuracy: 0.7515 - val_loss: 0.4899 - val_recall: 0.7641
Epoch 18/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7952 - loss: 0.4448 - recall: 0.8047 - val_accuracy: 0.7605 - val_loss: 0.4858 - val_recall: 0.7396
Epoch 19/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7947 - loss: 0.4398 - recall: 0.7934 - val_accuracy: 0.7565 - val_loss: 0.4778 - val_recall: 0.7592
Epoch 20/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7917 - loss: 0.4439 - recall: 0.7949 - val_accuracy: 0.7220 - val_loss: 0.5261 - val_recall: 0.7838
Epoch 21/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7925 - loss: 0.4473 - recall: 0.8039 - val_accuracy: 0.7680 - val_loss: 0.4620 - val_recall: 0.7322
Epoch 22/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7955 - loss: 0.4401 - recall: 0.7980 - val_accuracy: 0.7285 - val_loss: 0.5053 - val_recall: 0.7789
Epoch 23/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7921 - loss: 0.4388 - recall: 0.8005 - val_accuracy: 0.7505 - val_loss: 0.4946 - val_recall: 0.7690
Epoch 24/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7965 - loss: 0.4406 - recall: 0.8072 - val_accuracy: 0.7655 - val_loss: 0.4745 - val_recall: 0.7543
Epoch 25/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.7933 - loss: 0.4367 - recall: 0.7979 - val_accuracy: 0.7650 - val_loss: 0.4739 - val_recall: 0.7518
Epoch 26/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8005 - loss: 0.4259 - recall: 0.8074 - val_accuracy: 0.7590 - val_loss: 0.4688 - val_recall: 0.7297
Epoch 27/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.7982 - loss: 0.4330 - recall: 0.8033 - val_accuracy: 0.7560 - val_loss: 0.4882 - val_recall: 0.7666
Epoch 28/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7938 - loss: 0.4315 - recall: 0.7994 - val_accuracy: 0.7815 - val_loss: 0.4559 - val_recall: 0.7224
Epoch 29/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.7949 - loss: 0.4349 - recall: 0.7833 - val_accuracy: 0.7800 - val_loss: 0.4652 - val_recall: 0.7322
Epoch 30/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 4ms/step - accuracy: 0.8002 - loss: 0.4259 - recall: 0.7950 - val_accuracy: 0.7615 - val_loss: 0.4818 - val_recall: 0.7445
Epoch 31/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 5ms/step - accuracy: 0.8020 - loss: 0.4299 - recall: 0.8072 - val_accuracy: 0.7710 - val_loss: 0.4701 - val_recall: 0.7568
Epoch 32/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8026 - loss: 0.4310 - recall: 0.8084 - val_accuracy: 0.7865 - val_loss: 0.4545 - val_recall: 0.7174
Epoch 33/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8059 - loss: 0.4262 - recall: 0.8004 - val_accuracy: 0.7735 - val_loss: 0.4548 - val_recall: 0.7248
Epoch 34/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8025 - loss: 0.4240 - recall: 0.8105 - val_accuracy: 0.7820 - val_loss: 0.4650 - val_recall: 0.7273
Epoch 35/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8002 - loss: 0.4261 - recall: 0.8000 - val_accuracy: 0.7950 - val_loss: 0.4391 - val_recall: 0.7052
Epoch 36/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8078 - loss: 0.4227 - recall: 0.7995 - val_accuracy: 0.7740 - val_loss: 0.4668 - val_recall: 0.7322
Epoch 37/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.7999 - loss: 0.4266 - recall: 0.8071 - val_accuracy: 0.7865 - val_loss: 0.4514 - val_recall: 0.7125
Epoch 38/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 1s 3ms/step - accuracy: 0.8001 - loss: 0.4254 - recall: 0.8020 - val_accuracy: 0.7720 - val_loss: 0.4701 - val_recall: 0.7346
Epoch 39/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8053 - loss: 0.4255 - recall: 0.8074 - val_accuracy: 0.7825 - val_loss: 0.4549 - val_recall: 0.7125
Epoch 40/40
150/150 ━━━━━━━━━━━━━━━━━━━━ 0s 3ms/step - accuracy: 0.8115 - loss: 0.4202 - recall: 0.8111 - val_accuracy: 0.7850 - val_loss: 0.4453 - val_recall: 0.7027
In [ ]:
print("Time taken in seconds ",end-start)
Time taken in seconds  27.55156636238098
In [ ]:
plot(history,'loss')
In [ ]:
plot(history,'accuracy')
In [ ]:
plot(history,'recall')
In [ ]:
results.loc[5] = ['NN SMOTE with Adam/Dropout',2,'[64,32]','relu',epochs,batch_size,history.history["loss"][-1],history.history["val_loss"][-1],history.history["accuracy"][-1],history.history["val_accuracy"][-1],history.history["recall"][-1],history.history["val_recall"][-1],round(end-start,2)]
results = results.sort_index()
results
Out[ ]:
model label # hidden layers # neurons - hidden layer activation function - hidden layer # epochs batch size train loss validation loss train accuracy validation accuracy train recall validation recall time (secs)
0 NN with SGD 2 [64,32] relu 100 64 0.33 0.35 0.86 0.86 0.47 0.43 42.23
1 NN with Adam 2 [64,32] relu 40 64 0.31 0.37 0.87 0.85 0.51 0.45 20.51
2 NN with Adam/Dropout 2 [64,32] relu 40 64 0.35 0.36 0.86 0.85 0.43 0.42 20.63
3 NN SMOTE with SGD 2 [64,32] relu 100 64 0.40 0.60 0.82 0.70 0.82 0.84 68.58
4 NN SMOTE with Adam 2 [64,32] relu 40 64 0.32 0.56 0.86 0.76 0.88 0.74 29.96
5 NN SMOTE with Adam/Dropout 2 [64,32] relu 40 64 0.42 0.45 0.81 0.79 0.81 0.70 27.55

Commentary: This model provides the best balance of recall and not having excessive overfitting behavior.

Model Performance Comparison and Final Model Selection¶

Model 5, NN SMOTE with Adam/Dropout, offers the best performance in terms of providing an excellent recall score and also avoiding significant overfitting. Let's look at how this model performs on the test data.

In [ ]:
test_loss, test_accuracy, test_recall = model_5.evaluate(X_test, y_test, verbose=0)

# Printing Evaluation Results
print(f"Test Loss: {test_loss:.2f}")
print(f"Test Accuracy: {test_accuracy:.2f}")
print(f"Test Recall: {test_recall:.2f}")
Test Loss: 0.45
Test Accuracy: 0.79
Test Recall: 0.74

Print the confusion matrices for train, validation, and test.

In [ ]:
plot_confusion_matrix(model_5, X_train, y_train)
In [ ]:
plot_confusion_matrix(model_5, X_val, y_val)
In [ ]:
plot_confusion_matrix(model_5, X_test, y_test)

Actionable Insights and Business Recommendations¶

The selected model has a Test Accuracy of 79% indicating a decent ability to correctly predict the customer outcomes. Test Recall is 74%, so while a decent score, further work could be done to reduce the number of False Negatives (customers who actually churned, but were predicted not to churn). Unfortunately the Neural Network model behaves largely as black box, so it does not provide us insights directly as to the feature importances.

From the EDA, some observations:

  • About 20% of the customers churned.
  • As for the numeric variables in the data, there is almost no correlation between them.
  • There appears to be some increased liklihood of churning for customers around 50 years of age, +/- ~10 years.
  • Although a small number it appears that customers with 3 or 4 products have an elevated risk of churning.
  • Customers with very low (outlier) Credit Scores are at elevated risk of churning.

Power Ahead